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We, Ruth Rosenholtz, Allison G. Woodruff and Andrew Faulring, hereby declare and 
state that: 

1. This Declaration is submitted as evidence that the invention of this application 
was invented by us prior to June 15, 2001, which is the effective filing date of U.S. Patent 
Application Publication No. 2003/0006995 to Smith et al., entitled "Orthogonal Magnifier 
Within a Computer System Display," which was applied in the February 16, 2005 Office 
Action. 

2. We are the named inventors in the above-identified application. 

3. We are the inventors of the invention described in an invention proposal 
entitled "Enhanced Thumbnails of Documents," which appears as Exhibit A attached to this 
Declaration. The invention proposal is signed by us and dated October 3, 2000. 

4. The copies of these pages which constitute Exhibit A are true copies of the 
invention proposal. 



DECLARATION UNDER 37 C.F.R. SL131 



Commissioner for Patents 
P.O. Box 1450 

Alexandria, VA 22313-1450 



Sir: 



Xerox Docket No. D/A0A39Q 
Application No. 09/682,229 

5. The invention described by Exhibit A was conceived and actually reduced to 
practice by us in the United States at least as early as October 3, 2000. 

6. We were in possession of the invention recited in claims 1-3, 6-22, 25-28 and 
30-40 as evidenced by the entire disclosure of Exhibit A. 

7. We hereby declare and state that all statements made herein of our own 
knowledge are true t and that all statements made on information and belief are believed to be 
true; and further that these statements were made with the knowledge that willful false 
statements and the like so made are punishable by fine and/or imprisonment under Section 
1001 of Title 18 of the United States Code, and that such willful false statements may 
jeopardize the validity of the application or any patent issuing therefrom. 



Date: 

Ruth Rosenholtz 

Date: 



J Allison Gyle Woodruff 



Andrew Fauuhg 



Attachment: 

Exhibit A 



XEROX 



Protect Until: 




EXHIBIT A 



Invention Proposal 



THE DOCUMENT COMPANY 

XEROX 



Signed hard copy To: Xerox Intellectual Property Law Department IP A 0 1 5 0 

□ Xerox Square -20A, Rochester, NY 14644, MailStop XRX2-20A - Send electronic version to your mgr. & copy to: USA.IPLD.MC® mc.usa.xerox.com 

□ El Segundo.CA, 1 990 Xerox Centre Dr. 90245, MailStop ESC1 -405 -Send electronic version to your mgr. & copy to: 
USA.IPLD.ES@mc.usa.xerox.com 

B Palo Alto, CA, 3333 Coyote Hill Road 94304, MailStop: PARC - Send electronic version to your mgr. & copy to: USA.IPLD.PA@mc.usa.xerox.com 

□ Wilsonville,OR, 26600 SW Parkway 97070, MailStop: 7063-LAW Send electronic version to your mgr. & copy to: USAIPLD.OR@mc.usa.xerox.com 



1 


Proposal Submitted By (Please use legal name) Full First Name, Middle, Last 
Ruth Rosenholtz 


Employee No. 
989669 


Outside Phone No. 
(650) 812-4390 


Organization (Unit/Div7Dept7Section) 
XR&T/PARC/SPL/DiVA 


Electronic Mail Address 
rruth@parc.xerox.com 


Bldg. NoV Mail Stop 
35/1650 


Fax No. 

(650) 812-4334 


2 


Proposal Submitted By (Please use legal name) Full First Name, Middle, Last 
Allison Gvle Woodruff 


Employee No. 
994117 


Outside Phone No. 
(650)812-4429 


Organization (Unit/DivyDept ./Section) 
XR&T/PARC/ISTLAMR 


Electronic Mail Address 
woodruff@parc.xerox.com 


Bldg. NoV Mail Stop 
35/2330 


Fax No. 

(650) 812-4258 


*3 


Proposal Submitted By (Please use legal name) Full First Name, Middle, Last 
Andrew Faulrinq 


Employee No. 


Outside Phone No. 
(_)-_• 


Organization (Unrt/DivVDept ./Section) 
CMU (PARC summer intern) 


Electronic Mail Address 
faulring+@cs. cmu.edu 


Bldg. NoV Mail Stop 


Fax No. 



If space for additional submitters is required, please use another sheet; and attach any supplementar y Comments. 



Manager 
David Fleet 



Electronic Mail Address 
fleet @ parc.xerox.com 



Bldg. NoTMS 
35/1642 



Name of Xerox Program (if any) 



Technical Category 
(see attached list) 

Suggested categories: 1 .2, 2.4, 6.2 , _ — _ 

Opportunity for licensing revenue Who could be interested in it? How is this better than alternatives? 

Many search engine vendors could be interested in using this technology, among others. Our study shows that people 
are significantly more efficient searching web pages when using our enhanced thumbnails versus traditional text 
summaries or enhanced thumbnails. ■ _ 



Descriptive title of invention 

Enhanced thumbnails of documents 



Describe the problem How was this problem tackled before your invention? 

The challenge is to create thumbnails that are information-rich yet retain the ability to be scanned quickly. Previous 
thumbnails fall into two categories: (1) simple scale-reduced images of documents [e.g., 1,2,3] (occasionally augmented 
with metadata about the document [4]) and (2) images composed of selected elements or visual representations of 
metadata properties from the document [5]. The former contain little information, e.g., often none of the text in tho 
thumbnail can be read. The latter images bear little visual resemblance to the original document, so properties like genr© 
that can be scanned quickly are lost. It remains to be seen how quickly people can scan abstract representations of 
metadata. 



Summary of the invention Describe briefly what the invention is and how it works in 5 -8 lines. 

We introduce a technique for creating novel, enhanced thumbnails of documents. These thumbnaila combine \m 
advantages of image thumbnails and of other types of summaries (e.g. text), i.e., they can b© ecanntd quickh/ but contain 
more information than traditional thumbnails. Our thumbnails include a reduced imago of Ih© document Along with 
various forms of emphasis of information in the document: (1) modifying the document format (e.g., HTMl) prior to deal© 
reduction or (2) overlaying graphical elements (e.g., text callouts) on the scale-reduced thumbnail. 
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Describe your invention Describe how to make and use the invention and it's novel embodiments. Cover the process, method, materials with 
sketches, flow charts, usage etc. What are the advantages of your invention for Xerox? 

In this IP we describe our enhanced thumbnails, focusing on a particular instantiation of those thumbnails that we have 
tested and found to be more effective for a web search task than either text summaries or simple reduced-image 
thumbnails. We also describe, in brief, the system for generating these thumbnails, and in the process try to give a 
feeling for the breadth of what we mean by "enhanced thumbnails." For more details, we include our recent CHI 
submission, "Using Thumbnails to Search the Web/ 

Our enhanced thumbnails could be useful in virtually any system in which thumbnails might be useful. Some suggested 
applications are browsing or searching the web, management of email or an on-line document corpus, and so on. 
Though we have so far implemented a system that generates a thumbnail from an HTML document, one could generate 
enhanced thumbnails from virtually any text/mixed-media file format, and possibly even from an image format (e.g. a 
scanned document) plus OCR. 

Generating enhanced thumbnails is a multi-step process involving multiple renderings. The steps include the following, or 
some subset thereof: 

1 . Specify important elements of the document. 

2. Modify the original document format. 

3. Render a reduced image of the modified document. 

4. Extract properties of the reduced image. 

5. Re-render the thumbnails with one or more overlays, possibly making use of the extracted properties. 

Below we discuss these steps in somewhat more detail, giving examples of the resulting thumbnails. We believe 
thumbnails generated using such a 

series of steps to be novel, as they both allow a great deal of preservation of the original document format - which 
provides information about such things as document genre - and the presentation of additional information in the form of 
overlays on the basic thumbnails. Previous thumbnails have either been simple reduced images of a page of the 
document, possibly including abstract representations of document metadata, or abstract collections of elements or 
metadata from the document, represented in a way that loses all of the original appearance of the document. 

Specify important elements of the document- 
There are a number of ways that the important elements might be specified. The user might specify them, in some 
applications. In our implementation, which is a web search application, we took search terms and hi and h2 headers to 
indicate important text. One might also imagine using text words with high TF*IDF scores, terms resulting from 
information scent computations [6], the document title or URL, and so on. Items other than text might be labelled as 
important: images labelled as "representative" [5], tables, etc. 

In general, items could be specified as important elements by the output of any model of importance in a document: 
models of user intent, models of human perception, eye-tracking data, etc. 

Modify the format of the original document: 

Here we change the formatting of the original document, along any dimension in which one typically has control of the 
formatting. In our implementation we increase the size of hi headers so as to make them "readable" in the reduced 
thumbnail image (we specify the size required for readability by hand, but this might also be done automatically in various 
ways). We also highlight important words specified by the previous step. 

Render a reduced-size version of the modified document: 

In our implementation, we render a simple reduced image of either the first page of the document, a page of the 
document starting at an arbitrary point in the document, or the full document. Among other variants one might imagine 
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rendering with an arbitrary affine transformation or distortion of the document [8]. This step could make use of prior art on 
better filtering and subsampling techniques for generating reduced document images [e.g. 9]. 

Extract properties of the reduced image: 

In our implementation, this includes extracting statistics of the colors in the reduced image, and the location and size of 
selected elements within the reduced image. 

Re-render the thumbnails with one or more overlays: 

In our implementation, we first overlay a translucent "fill" color over the entire thumbnail. This desaturates the thumbnail 
colors. This decreases the distraction from attention-grabbing elements in the original document, allowing our overlays to 
better draw attention, yet still allowing the basic structure of the original document to show through. We also overlay 
enlarged words from step 1 , centered on the location of the same text in the underlying thumbnail. In our implementation, 
this text is highlighted by one of several translucent colors, each word consistently highlighted with the same color. We 
selected the highlight colors in a principled way, based upon a model of human vision, but they might also be selected by 
hand or in another way. Our implementation allows these highlighted words to (optionally) extend over the edge of the 
thumbnail. Furthermore, in our implementation these overlaid words maintain their original format except where specified, 
so that, e.g., HTML links still look like links (blue, underlined). 

Figure 1 shows a number of examples of thumbnails we have generated that satisfy this constructive definition of 
enhanced thumbnails. As described in the paper, we have tested one variant of enhanced thumbnails in a web search 
task, comparing with performance using standard text summaries or plain reduced-image thumbnails. We found that 
users were significantly faster at finding the answer to their query using the enhanced thumbnails. Because of this result 
and our own experiences using the thumbnails, we believe that at the very least this invention may provide a licensing 
opportunity for Xerox to some web search company. 

One area we have not yet explored is the use of animation in the thumbnails. For example, the thumbnails might animate 
to emphasize different pieces of information at different times, e.g., on mouse-over. 

In addition to the noveity of the enhanced thumbnails, we believe there may be novelty in a couple of navigation system 
improvements suggested by the enhanced thumbnails. We have implemented neither of these system enhancements. 

We call the first "click-through. 0 Normally, when a user clicks with a mouse on a thumbnail, we'd like them to go to the 
document represented by the thumbnail. With click-through, the user would click in a particular way on a particular part of 
the thumbnail, and would either be brought tolhe corresponding part of the document, or would be brought to a document 
linked to by the corresponding link in the original document. Which of the two actions occurred might depend both on 
where the user clicked (is there a link in that location?) and how they clicked (middle button vs. right, keypress or no, etc.). 
This system enhancement is enabled by the enhanced thumbnails, since they may enhance text that will guide the user to 
a particular location in the document, and also may enhance links so that they are visible, obviously links, and large 
enough to click on easily. 

The second system enhancement suggested by our thumbnails is also to enhance the document, at least temporarily. In 
this addition to the system, a user might click on an enhanced thumbnail and be taken to a document enhanced in the 
same way. This consistency between thumbnail and destination document should help the user find the document 
elements that drew their interest in the thumbnail. The enhanced document might, after a short time, morph to the 
original document. 
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Figure 1. Top row (from left to right): Plain thumbnail. Thumbnail enhanced with HTML modification. Thumbnail enhanced with HTML 
and image modification. E-commerce genre example. Bottom row (from left to right): News genre example. Homepage genre 
example. Plain thumbnail of text page. Enhanced thumbnail of text page. 
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People List names of others known to have worked on this or a similar invention 
Examples of plain thumbnails: 

[1] Ayers, E. and Stasko, J. Using Graphic History in Browsing the World Wide Web. in Proc. 4th Intl. WWWConf., 
Boston, December 1995. 

[2] Hightower, R., Ring, L, Helfman, J., Bederson, B., and Hollan, J. Graphical Multiscale Web Histories: A Study of 
PadPrints. In Proc, Hypertext'98, 1998, 58-65. 

[3] Kopetzky, T. and Muhlhauser, M. Visual Preview for Link Traversal on the WWW. in Proc. 8th Intl. WWW Conference, 
Toronto, Canada, May 1999, 447-454. 

Plain thumbnails plus metadata: 

[4] Cockburn, A., et al. WebView: A Graphical Aid for Revisiting Web Pages. In Proc. OZCHI'99 Australian Conf. on HCI, 
Wagga Wagga, Australia, Nov. 1 999. 

Thumbnail "caricatures" - selected elements plus metadata, not attempting to mimic the appearance of the document: 

[5] Wynblatt, M. and Benson, D. Web Page Caricatures: Multimedia Summaries for WWW Documents. In Proc. IEEE Intl. 
Conf. on Multimedia Computing and Systems, Austin, TX, June, 1998, 194-199. 

Information scent: 

[6].OIston, C. and Chi, E. ScentTraiis: Integrating Browsing and Searching on the World Wide Web, Submitted to 
SIGCHI 2001. 

Fish- eye views: 

[7] Card, S. K., Mackinlay, J. D. t and Shneiderman, B. Information Visualization: Using Vision to Think. Morgan- 
Kaufmann, San Francisco, 1998. 

[8] Patent #5740285, Bloomberg, Dan S. and Davies, Daniel. Image reduction/enlargement technique. 

Related concepts Check the Xerox Patent data base at httD://comiD.wrc.xerOX.COm/COmiD/icbuhome.nsf 

What have you found in a data base search of the topic? Give patent or IP number of the most relevant items. 
Patent #5526443, Nakayama, Takehiro. Method and apparatus for highlighting and categorizing documents using coded 
word tokens. 

Patent #5384863, Huttenlocher, Daniel P. et al. Methods and apparatus for automatic modification of semantically 
significant portions of a document without document image decoding. 

Patent #5060135, Levine et. al. Apparatus for manipulating documents in a data processing system utilizing reduced 

images of sheets of information which are movable. 

Prototype Has a model, a prototype, or experiment of the invention been built, made, run or tested? ]S Yes □ No 

Yes. A prototype has been written in Java, using a component web browser (ICE Browser 5). 



Xerox product Is the invention used by Xerox or is there a definite plan for use in a future product(s) ? □ Yes H No 

If so, please identify the program(s) or product(s), and introduction dates: 
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Disclosures Has this concept been disclosed to vendors, consultants, outside parties, partners, etc? Indicate the date(s) of any previous or planned 
future disclosure external to Xerox, and identify the type of disclosure (by agreement, demonstration, paper or presentation given, market 
probe, published article, etc., and if convenient, please provide a copy of the agreement, paper or article): 



Outside funding & YES (Indicate Source of outside funding) □ NO 

^es^Pactial funding for portions of the project were received by Office of N No. : N0O0i4-96-e-^* 
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Document Systems Architecture 


2.4 


Representation 
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Electronic Document Commerce 
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Development (Re-lmageable Process) 




Process) 
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Submitter(s): 



Title of Invention 



Manager's Name 



Date 



1 . Problem addressed or function provided by the invention: 
Example 1A: Finisher cost reduction 



Example 1B: Uses low cost LCD to write annotation messages 



2. Central thrust of the invention: 

Example 2A: Design incorporates fewer parts 



Example 2B: Uses low cost LCD to write annotation messages 



3. Could invention have impact beyond current description? 
Example 3A: Could also function for printer finisher 



Example 3B: Could also function to erase/edit copy 



4. Potential for Xerox application. Specify product or technology program if possible: 



Example 4A: Mainline approach in Program Q 



Example 4B: Adds significant feature to future products 



5. Value to competitors; potential for license or trade: 

Example 5A: Enables much lower cost finishing than any known system Example SB: Could be licensed in a business area un-related to Xerox 
and opens possibilities of moving finishing down-market 



6. Please indicate any related patents, publications, or activities you know of: 



7. I would recommend the following form{s) of protection: □ Patent 



□ Defense publication □ Keep trade secret □ None 
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